Draft
Conversation
…dd other selectors to prep_sliiders, allow other name options for dims of input stores, improve performance of create_template_dataarray, drop now unnecessary save_to_zarr_region wrapper
…k for surge lookup completeness to save time, allow chunking along other dims than seg, use as_completed instead of client.gather, add some print stmts for model status, fix use of scen_mc_filter in execute_pyciam, remove erroneously left testing return in execute_pyciam, allow for unique spec of quantile to use for refA, drop custom save_to_zarr_region func
…max of slr data; improve dtypes of template zarrs;
There was a problem hiding this comment.
Pull request overview
This PR updates pyCIAM to handle large Monte Carlo ensembles of sea level rise projections, addressing several bugs and performance issues that arose when running in this probabilistic context.
Key changes:
- Enhanced flexibility for handling variable base years, dimension names, and subset selections across SLR and economic input data
- Improved error resilience and memory efficiency in surge lookup table creation
- Migration from custom
save_to_zarr_regionto xarray's nativeregion="auto"parameter
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
| pyCIAM/utils.py | Added support for variable base years in spatial data, made seg_var and mc_dim parameters configurable, added documentation for refA |
| pyCIAM/surge/lookup.py | Enhanced surge lookup creation with configurable dimension names, improved dtype handling for Zarr 3 compatibility, added workflow completion checks |
| pyCIAM/run.py | Major refactoring to support flexible chunking, improved workflow tracking with as_completed, enhanced parameter handling for large ensembles |
| pyCIAM/io.py | Refactored SLR data loading to handle quantile computation on-the-fly, improved memory efficiency, updated selector handling |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thus far, pyCIAM has only been run on a couple quantiles of sea level rise per segment. In theory, it can be run on large ensembles of monte carlo SLR projections, as needed for probabilistic estimates of global damages (as used for SC-GHG calculations).
The last time that pyCIAM was run in this context was prior to the full development of the package, and the package is not fully set up to handle this context. This PR contains a number of updates enabling the execution in this large ensemble context (as well as some others)
scen_mc_filterto specify only running certain sceanrios and monte carlo samples